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4.0 Executive Summary 

The rise of innovative unmanned aeronautical systems and the emergence of commercial space 
activities have resulted in a number of relatively new aerospace organizations that are designing 
innovative systems and solutions. These organizations use a variety of commercial off-the-shelf 
and in-house-developed simulation and analysis tools including 6-degree-of-freedom (6-DOF) 
flight simulation tools. The increased affordability of computing capability has made high- 
fidelity flight simulation practical for all participants. 

Verification of the tools’ equations-of-motion and environment models (e.g., atmosphere, 
gravitation, and geodesy) is desirable to assure accuracy of results. However, aside from simple 
textbook examples, minimal verification data exists in open literature for 6-DOF flight 
simulation problems. 

This assessment compared multiple solution trajectories to a set of verification check-cases that 
covered atmospheric and exo-atmospheric (i.e., orbital) flight. Each scenario consisted of pre- 
defined flight vehicles, initial conditions, and maneuvers. These scenarios were implemented 
and executed in a variety of analytical and real-time simulation tools. This tool-set included 
simulation tools in a variety of programming languages based on modified Hat-Earth, round- 
Earth, and rotating oblate spheroidal Earth geodesy and gravitation models, and independently 
derived equations-of-motion and propagation techniques. The resulting simulated parameter 
trajectories were compared by over-plotting and difference-plotting to yield a family of 
solutions. In total, seven simulation tools were exercised. 

Participating in the assessment were participants from NASA Ames Research Center (ARC), 
Armstrong Flight Research Center (AFRC), Johnson Space Center (JSC), Langley Research 
Center (LaRC), and Marshall Space Flight Center (MSFC), and an open-source simulation tool 
development project (i.e., JSBSim). 

The vehicle models were published in the American Institute of Aeronautics and Astronautics 
American National Standards Institute (AIAA/ANSI) S-l 19-201 1 Flight Dynamics Model 
Exchange Standard [ref. 4] markup language, making them realizable in a variety of proprietary 
and non-proprietary implementations. This set of models and the resulting trajectory plots from 
a collection of simulation tools may serve as a preliminary verification aide for organizations 
that are developing their own atmospheric and orbital simulation tools and frameworks. 

This document is an overview of the process used and the results of the assessment. Volume II 
contains details on models, implementation, and results. Simulations from atmospheric check 
cases found the following: 

• Minor differences in results from tabular versus equation-based atmosphere models, and 
geodetic versus geocentric geometries; 

• To a smaller degree, some differences in the implementation of the square-law and 
harmonic gravitation are also apparent due to differences in gravitation model 
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implementation or in the conversion of the initial geodetic position into the geocentric 
position (since position is an input into the gravitation model); and 

• Differences in numerical integration methods in the different simulation tools appeared to 
cause some differences in predicted trajectories. 

Comparison of simulations from the orbital check cases showed good agreement; as in the 
atmospheric check cases, differences in numerical integration methods appeared to cause minor 
differences in predicted trajectories. 

In general, the simulation comparisons showed good agreement. Coding errors, incorrect initial 
conditions, and invalid assumptions were discovered and corrected as a result of this comparison 
exercise. Differences between simulation tools in implementation of gravitation, atmospheric, 
and Earth orientation models remain, which require further effort to resolve. 

Not all proposed scenarios were successfully implemented on a sufficient number of independent 
simulation frameworks (i.e., established minimum number of three) to warrant inclusion in this 
assessment. However, it is hoped these scenarios can be contributed by other organizations and 
included in the public repository of results. 

The models and data are available from the NASA Engineering and Safety Center’s (NESC) 
Academy website, in the Flight Mechanics area [ref. 7]. 

This assessment is believed to be the first publically available comparison of a set of 6-DOF 
flight simulation tools. An earlier NASA study [ref. 1] compared exo-atmospheric scenarios 
between NASA and international space agency partners, but those results are not publically 
available. 

Lessons learned in this exercise, given in findings, observations, and NESC recommendations, 
might prove useful in future simulation tool development. Among these is an identified need for 
a convention regarding initial conditions and an improved method for sharing time-history data. 
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5.0 Assessment Plan 

The NASA Technical Fellow for Flight Mechanics assembled an assessment team to develop 
verification data sets. This team met at LaRC to map out an approach to developing check-cases 
for comparison and cross-verification purposes. 

The assessment team agreed a set of scenarios involving simple models would be developed and 
simulated by each participant in their preferred simulation tool. The basic parameters were 
agreed upon and further discussion led to the set of scenarios described in Section 6.2.8. 

Formats for specifying the models, initial conditions, and resulting time-history data were agreed 
to and a plan for presenting the data were developed. 

Instead of identifying a single “known good” simulation tool, or requiring all trajectories match 
within a predefined tolerance, the approach taken was to present comparison plots of the results 
of each simulation tools. If acceptable agreement between the parameter trajectories generated 
by the tools was found, then those trajectories could serve as a verification guide. If 
unacceptable difference in results was evident, then an attempt would be made to identify an 
assumption, design choice, and/or an implementation difference to explain the disparity, and the 
set of trajectories would serve as a family of possible solutions. 

One of the overall objectives was to generate a publically available report containing the salient 
results for use by current and future organizations. 

6.0 Problem Description, Proposed Solutions, and Risk Assessment 

6.1 Problem Description 

The independently developed NASA, industry, and commercial flight simulation tools in use for 
flight dynamics and trajectory predictions have sometimes provided substantially different 
results. Some of the disagreements have been traced to differences in equations of motion 
(i.e., kinematics) implementation and the geodetic, gravitational, and atmosphere models. 
Differences have been caused by the levels of precision used for physical constants, and 
inconsistent interpretations of how to implement and initialize a given scenario. Other sources of 
differences have arisen from inconsistent or limited-precision unit conversions. 

At the start of this assessment, there were no accepted benchmark check-cases that could be used 
for verification of a simulation tool. This led to the risk in using unverified tools for flight 
prediction and design in support of NASA flight projects. Due to the non-linear nature of most 
simulation scenarios, an analytical (i.e., closed-form) solution of the resulting trajectory is rarely 
available. The work-around solution has historically been running similar but independently 
developed simulations of specific flight vehicles and working to resolve differences between the 
preflight simulations, and later using actual flight data to improve simulation model fidelity. 
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6.2 Proposed Solution 

In an attempt to build a “consensus” solution for 6-DOF flight vehicle simulations, a set of 
relatively simple flight vehicle models was developed, with a set of maneuvers from specified 
initial conditions, in a variety of atmospheric, gravitational, and geodetic configurations. It was 
anticipated the resulting trajectories would fall into one or more families of solutions based upon 
assumptions and simplifications (e.g., flat-Earth conditions). 

It was desirable to make the vehicle models and resulting trajectory data available electronically 
for ease of comparison by developers of other simulation tools. 

6.2.1 Check-Case Vehicle Models 

A set of reference flight vehicles was proposed, based primarily on existing non-proprietary 
vehicle models, which are described in detail in Appendix B.l. 

For the atmospheric scenarios, the “vehicles” included: a spheroid (i.e., cannon ball); a brick to 
evaluate rotational dynamics; a subsonic fighter with representative nonlinear aerodynamics, 
propulsion, and control law models; and a two-stage rocket. For the orbital cases, a larger 
spheroid, a cylindrical rocket body, and a simplified International Space Station were re-used 
from an earlier comparison study. 

6.2.2 Check-Case Geodesy Models 

One of the challenges in performing 6-DOF flight simulations is the choice in how to model the 
Earth’s shape and motion. Early low-speed atmospheric flight simulations often used a flat- 
Earth approximation, which was sufficient for recreating landing and takeoff dynamics. Early 
computational performance limitations made this simplifying approximation attractive for pilot- 
in-the-loop (“real-time”) training or research and development simulations. 

As digital computers grew in capability, simulation of flight using more complex spherical and 
oblate rotating Earth models became practical from a cost/time standpoint. Many atmospheric 
flight simulation tools incorporate the standard DoD World Geodetic System 1984 (WGS-84) 
[ref. 2] ellipsoidal Earth model even though an iterative solver, or other multi-step iterative 
process, is normally required to convert between inertial coordinates and geodetic coordinates 
(i.e., latitude, longitude, and altitude) with the ellipsoidal geodesy model. 

The atmospheric check-case scenarios developed for this study included round non-rotating, 
round rotating, and oblate spheroidal rotating Earth models. The orbital check-case scenarios 
used the oblate WGS-84 model exclusively. More information on each geodesy model is 
described in Appendix B.2. 

6.2.3 Check-Case Coordinate Systems 

A number of coordinate system definitions and transfonnations were required in this assessment 
including: J2000 inertial; Earth-centered inertial (ECI); Earth-centered Earth-fixed (ECEF) in 
either geocentric or geodetic frames; local-vertical, local-horizontal (LVLH); north-east-down 
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(NED); runway; and body coordinates. These systems and transformations are discussed in 
Appendix B.3. 

6.2.4 Check-Case Gravitation Models 

In parallel with a choice of Earth geodesy models is a corresponding choice of gravitation 
models. The simplest model has gravitational attraction varying inversely with the square of the 
vehicle distance from Earth’s center. This simplified model is often used with the approximation 
of a spherical Earth. 

A more sophisticated gravitation model, including gravitational harmonics that vary with latitude 
and longitude, is normally employed for ellipsoidal Earth models. For atmospheric check-cases 
with a WGS-84 Earth, the first non-zero term of the harmonic series (i.e., Ji gravitation) is 
included. Orbital scenarios included the Ji and higher harmonic terms (i.e., to 8 x 8). More 
information on gravitation models is described in Appendix B.4. 

6.2.5 Check-Case Atmosphere Models 

US 1976. The US Standard 1976 Atmosphere model [ref. 3] was used for the majority of the 
atmospheric check-case scenarios. This model can be implemented as linear interpolation of the 
one-dimensional tables given in the source document with ambient pressure, temperature, and 
density as a function of geometric altitude ( h ) or geopotential height (Z). A more accurate 
implementation is to realize the non-linear numerical equations from reference 3 used to generate 
the tables published in the reference. 

Marshall Engineering Thermosphere (MET). The MET is appropriate for modeling the 
thermosphere region of the Earth’s atmosphere, located above the stratosphere (i.e., greater than 
90 km) and below the exosphere (i.e., less than 500 km). MET is employed for most of the 
orbital check-cases. This model is not publicly available, but can be requested from the MSFC 
Natural Environments Branch. Details on MET are in Appendix B.5. 

6.2.6 Check-Case Data Formats 

The use of standard formats should significantly shorten the process of sharing models and 
comparing results. While some setup was required for each tool to receive models in an 
unfamiliar format and translate the data in a locally-compatible format, it was hoped the ability 
to quickly implement model changes and generate new results would be enhanced by this 
investment. 

Reference models. Most of the atmospheric check-cases vehicle models were specified using 
the format in reference 4 (i.e., S-l 19), which makes use of an extensible markup language 
(XML) based grammar, DAVE-ML [ref. 5], This document attempted to define the salient flight 
characteristics of an aerospace vehicle (i.e., aerodynamics and inertia) in an unambiguous text 
file that is human- and machine-readable, and with sufficient metadata to be easily converted 
into code and readily archivable. The most complex model attempted in this study was the 
single-engine F-16 aircraft defined in DAVE-ML using S-l 19 variable names that included an 
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inertial/mass properties model, a non-linear aerodynamic model, and two separate control law 
subsystem models. These models are available from reference 7 and described in Appendix B.l. 

Time-History Data. Despite an attempt to identify a more efficient binary data format for the 
several million data points that were generated in this effort, the assessment team stored data in a 
comma-separated-values (.CSV) text format. These files used column headers to identify the 
values represented and rows to group values associated with regular time steps of simulation. 

The check-case files were large (e.g., 12.MB in one case) as a result of using text instead of 
binary value representations, but this format was felt to be better suited for archival purposes and 
to be more readily accessible by other reviewers. The time-history data files are available from 

http://nescacademv.nasa.gov/flightsim/index.html 

6.2.7 Participating Simulation Tools 

Developers of several NASA and one open-source simulation tools agreed to participate in this 
comparison on a voluntary basis. The set of tools involved included simulations suited primarily 
for atmospheric flight, exo-atmospheric flight, and some were applicable to both flight regimes. 

Not all tools attempted to execute every check-case. The assessment team set a ground rule that 
a minimum of three data sets (i.e., parameter trajectories) were necessary to warrant inclusion in 
this assessment. 

The assembled tool-set included: 

• Core from ARFC 

• JEOD from JSC 

• JSBSim [ref. 6] 

• LaSRS++ from LaRC 

• MAVERIC from MSFC 

• POST-II from LaRC 

• VMSRTE from ARC 

Details on each tool are found in Appendix B.6. 

6.2.8 Check-Case Scenarios 

A set of atmospheric and orbital flight scenarios, models, and initial conditions was developed by 
the assessment team (see Tables 6.3-1 and 6.3-2). Details on each atmospheric and orbital flight 
scenarios can be found in Appendices C.l and C.2, respectively. 

Seventeen atmospheric check cases were identified and sixteen cases were run with at least three 
simulation tools (one case, number 14, was also run by three simulation tools, but was not 
included in the assessment due to lack of agreement on test inputs). Twenty-six orbital check 
cases were identified and all were run with at least three simulation tools. 
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6.3 Known Risks and Mitigations 

For this assessment, the risk of accidental agreement arising from a common error in predicted 
trajectories obtained from established, independently developed, rigorously tested, simulation 
tools appears unlikely. However, the consequence of a common error remaining undetected 
could be consequential. Sharing the results of this assessment with the wider aerospace 
community is intended to allow others to compare their tools against the ones represented here to 
minimize this risk. 
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Table 6.3-1. Atmospheric Check-Case Scenarios 


Number 

Name 

Verifies 

Gravitational 

Geodesy 

Winds 

1 

Dropped sphere 
with no drag 

Gravitation, 

translational 

EOM 

Ji 

WGS-84 

Still air 

2 

Tumbling brick 
with no 
damping, no 
drag 

Rotational EOM 

Ji 

WGS-84 

Still air 

3 

Tumbling brick 
with dynamic 
damping, no 
drag 

Inertial coupling 

Ji 

WGS-84 

Still air 

4 

Dropped sphere 
with constant 
Cd, no wind 

Gravitation, 

integration 

MR 2 

Round 

fixed 

Still air 

5 

Dropped sphere 
with constant 
Cd, no wind 

Earth rotation 

MR 1 

Round 

rotating 

Still air 

6 

Dropped sphere 
with constant 
Cd, no wind 

Ellipsoidal Earth 

Ji 

WGS-84 

Still air 

7 

Dropped sphere 
with constant 
Cd + wind 

Wind effects 

Ji 

WGS-84 

Steady 

wind 

8 

Dropped sphere 
with constant 
Cd + wind 
shear 

2 dimensional 
wind 

Ji 

WGS-84 

Ah) 

9 

Sphere 
launched 
eastward along 
equator 

Translational 

EOM 

Ji 

WGS-84 

Still air 

10 

Sphere 
launched 
northward 
along prime 
meridian 

Coriolis 

Ji 

WGS-84 

Still air 
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Number 

Name 

Verifies 

Gravitational 

Geodesy 

Winds 

11 

Subsonic F-16 
trimmed flight 
across planet 

Atmosphere, air- 
data calculations 

Ji 

WGS-84 

Still air 

12 

Supersonic F- 
16 trimmed 
flight across 
planet 

Supersonic air- 
data calculations 

Ji 

WGS-84 

Still air 

13 

Subsonic F-16 

maneuvering 

flight 

Multidimensional 
table look-up 

Ji 

WGS-84 

Still air 

14 

Supersonic F- 
16 

maneuvering 
flight (not 
completed) 

Mach effects in 
tables 

Ji 

WGS-84 

Still air 

15 

Circular F-16 
flight around 
North pole 

Propagation, 

geodetic 

transforms 

Ji 

WGS-84 

Still air 

16 

Circular F-16 
flight around 
equator/dateline 
intersection 

Sign changes in 
latitude and 
longitude 

Ji 

WGS-84 

Still air 

17 

Two-stage 
rocket to orbit 

Staging, entire 
atmosphere 

Ji 

WGS-84 

Ah) 
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7.0 Assessment Results 

A case-by-case comparative analysis of each of the check-case trajectories is given in Appendix 
D. A summary of the results is provided in the following sections separated between 
atmospheric and orbital regimes. 

A ground rule used by the assessment team in providing comparisons was that at least three 
simulation tools had to submit results for each check-case included in these results. Additional 
planned cases (e.g., supersonic fighter maneuvering flight and a proposed Apollo-like capsule 
reentry) were not included due to an insufficient number of implementations achieved. A total of 
16 of the 17 atmospheric check cases were completed and all of the 27 orbit check cases were 
completed. 

7.1 Atmospheric check-case results 

In general, comparisons of the atmospheric check-cases as simulated by several simulation tools 
indicate minor differences due to two variations in implementation: tabular versus equation- 
based atmosphere models, and geodetic versus geocentric geometries. 

In earlier computationally-constrained simulation implementations, an atmosphere model 
(e.g., reference 3 employed for these atmospheric flight simulations) was implemented as a table 
of density, temperature, and pressure values as a function of geometric height above a reference 
surface. This table was used in a linear interpolation between altitudes since this was typically 
faster than performing the complex calculations necessary to determine these quantities 
algebraically. 

Improved processors have made the direct calculation approach economically feasible and more 
precise. However, several of the participating simulation tools continue to use an atmospheric 
table implementation. Therefore, some of the trajectory differences are due to linear 
interpolation of atmospheric properties. 

The other main difference between results in atmospheric comparisons is an artifact of historical 
simulation techniques. As mentioned, earlier digital flight simulations of subsonic aircraft often 
assumed a flat Earth, where latitude and longitude were directly related to a Cartesian grid in the 
vicinity of a runway or airport. This was an appropriate approximation for low-speed flight in 
the vicinity of and while maneuvering around the terminal environment. Since the check-cases 
specified at least a round Earth, some retrofitting was undertaken to adapt the flat-Earth 
approximations to a round or oblate Earth. However, some artifacts of the simpler geodesy 
assumption remain, which are noted in Appendix D. 

To a smaller degree, some variances in the implementation of the square-law and harmonic 
gravitation were due to differences in gravitation model implementation, or in the conversion of 
the initial geodetic position into the geocentric position. Another variance source in the F-16 
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check-cases was differences in defining the equilibrium (i.e., trim) values for straight and level 
flight, especially the trimmed rotational rate. 

Errors in participating simulation tools that were initially uncovered, but corrected included 
mistakes in gravitational models, incorrect or imprecise initial condition values and geophysical 
constants, a one-frame time shift in gravitational value, and a transposition error in atmospheric 
property tables. For example, one simulation routinely and incorrectly aligned gravitational 
attraction along the geocentric radius axis, not the geodetic nadir. This led to a very, very small 
difference in the resulting trajectories that might not have been quickly identified without this 
assessment. 

Finally, differences in numerical integration methods in the simulation tools appeared to cause 
trajectory differences. These differences are hypothesized, as no specification of (or sufficient 
data regarding) integration techniques was available. 

In all cases, these differences were minor with the comparison plots found in Appendix D. Only 
when plotting variances between individual simulation results versus consensus or averaged 
results are the differences apparent. 

It should be noted that obtaining correlation between these simulations was an iterative process. 
Initial results were not as good as those ultimately obtained due to ambiguity in specification or 
implementation of initial conditions, maneuver inputs, and other simulation implementation 
differences. 

A total of 84 trajectories were generated comprised of nearly four million data points; these data 
sets are stored in 64 MB of data files available in the repository [ref. 7]. 

7.2 Orbital Check-Case Results 

Comparison of orbital check-cases showed good comparisons with few significant differences. 
As with the atmospheric cases, some iteration was required as significantly different results were 
initially obtained. These differences included use of different revisions of the MET model, 
differences in the specification of the Earth’s position at the start of various scenarios, 
differences in integration technique, or to mis-interpreting a sign convention or initial condition 
specification. 

An error was discovered (and corrected) in one of participating simulation tools in which an 
external force or moment was applied for a length of time other than what was specified in the 
configuration. This error was introduced in a recent rewrite of that particular module of the 
simulation tool and had somehow managed to elude detection, despite extensive regression tests 
that are routinely applied to all revisions. The revised tool had not yet been released, but the 
error may have affected NASA missions if it had not been detected during this assessment. The 
tool architect stated that he believed this ‘catch’ was worth the cost of the assessment. 

A total of 103 trajectories were generated comprised of 1 .4 million data points. These data sets 
are stored in 25 MB of data files available in the repository [ref. 7]. 
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7.3 Comparison Difficulties 

During this assessment, it became apparent that the time required to reach a reasonable level of 
match had been underestimated. The original schedule developed and agreed to by the team 
expected to complete this effort in just over 12 months. The effort has, after 30 months, not been 
completed to the degree expected at the outset, in that one atmospheric check-case (Earth reentry 
from a lunar return trajectory) has not been attempted, and a second atmospheric case remains 
incomplete. 

Part of the delay was due to the now-apparent need to specify initial conditions and maneuvering 
inputs exactly. It was believed early in the planning process that it would be sufficient for the 
scenarios to be described briefly in one axis frame; however, obtaining good matches ultimately 
required detailed specification of the initial conditions in several axis frames. An example is the 
initial rotation rate for some of the early atmospheric check-cases: a small numeric difference 
exists between the inertial and the ECEF angular rate of a body. Ensuring close matches 
required giving the rotation rate in both frames to ensure all simulation tools started with the 
same rate, since some simulation tools are initialized in ECEF-relative rates and others in inertial 
rates. A complete description of the changes and corrections required to improve matching is 
given in Appendix E, Section E. 1 . 

As knowledge was gained in this process, the initial conditions document had to be revised 
several times, initially leading to confusion by the team on which version was used in each round 
of comparison plots, which delayed reaching successful matches. 

The process followed by the widely dispersed team also introduced delays. Due to the large 
amount of data involved, considerable time was spent uploading data sets from each tool to a 
central server, downloading and plotting the trajectories by one analyst, uploading the results, 
and downloading and inspecting the large number of resulting plots for differences. Obvious 
differences were fairly easy to detect, but detennining the root cause of the difference often took 
considerable time and effort. 

A formal comparison (comprising Appendix D) by one analyst required a period of several 
weeks, due to the large number of maneuvers to compare and the in-depth analysis required. 

Since most participants were not full-time on this assessment, some of these comparison cycles 
took longer than others due to Agency priorities. Many more comparison cycles were also 
required than originally expected (30 sets of comparison plots were generated for the 
atmospheric cases between May 2013 and August 2014). 

7.4 Summary of Comparisons 

The eventual matches between simulation tools, achieved only after several iterations of 
comparing results and correcting mistaken assumptions and other errors, was good enough to 
indicate agreement between a majority of simulation tools for all cases published. Most of the 
remaining differences are explained and could be reduced with further effort. 
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Due to differences in trim algorithms, some of the 6-DOF aircraft check-cases (cases 10-16) 
provide some remaining disagreements on precise numbers, but do indicate a family of solutions 
that are close enough to serve as a comparison with other simulation tools. 

The orbital cases agree quite well. The remaining differences are attributed to either an obvious 
misconfiguration of the simulation tool or differences in the numerical integration method and 
step size. 

Appendix E gives quantitative measures of matches for each check-case, as well as some ideas 
for future work. If additional matching were to be funded, the authors welcome participation by 
other simulation tool developers who might wish to participate. 

8.0 Findings, Observations, and NESC Recommendations 

8.1 Findings 

The following findings were identified related to the simulation process: 

F-l. Even modeling simple vehicles posed challenges. Differences in the implementation of 
simple vehicle models were apparent. 

• These arose primarily from differences in interpretation of the scenario and initial 
conditions. 

• Initial attempts to model these scenarios led to some significant miscompares that 
revealed differences in physical constants and other modeling errors. These 
differences in constants and modeling errors were corrected in several tools prior to 
generating this set of comparisons. 

F-2. It took significant effort to get good agreement on the check-cases. Even simple 
aerospace vehicle simulation models are non-trivial to implement. 

• The comparable results shown in most of these check-cases required extensive 
iterations and adjustments/corrections to initial conditions and modeling assumptions. 

The following findings were identified related to atmospheric simulations: 

F-3. Tabular versus equation-based atmosphere introduced differences. 

F-4. Different interpretations of nadir direction can lead to differences due to “non-vertical” 
gravitational residue. 

F-5. While the majority of atmospheric simulations appeared to use WGS-84 Earth geodesy, 
simplifications such as flat- or round-Earth led to differences. 

F-6. Precise specification of initial conditions would be assisted by a standard for specifying 
the state vector of a 6-DOF flight simulation. 
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F-7. Every simulation examined eventually matched trajectories with at least two other 
simulations to a reasonable degree, where the correlation level is a function of the 
simulation purpose. 

The following finding was identified for the orbital simulations: 

F-8. In general, the orbital cases (implemented in at least three different simulation tools) 
matched fairly well, but minor differences are apparent. 

The following general findings were identified: 

F-9. In general, the atmospheric cases do not match as well as the orbital cases. 

• Atmospheric flight is non-linear, due to forces and moments being related to the 
square of the air-relative vehicle velocity, and to other non-linear aerodynamic 
effects. 

• The larger number of simulation tools were applied to these initial cases increased the 
chances of mismatches. 

F-10. The amount of effort required to develop, specify, and reconcile differences for multiple 
vehicle models across an array of simulation tools was grossly underestimated. 

F-ll. The comparison check-cases examined form the basis of a comprehensive set of 

verification data sets for 6-DOF flight simulations. Additional scenarios and results 
would improve the value of this process. 

• Comparison cases are needed for supersonic maneuvering flight and atmospheric re- 
entry scenarios. 

F-12. Nearly every simulation framework that participated in this assessment discovered at 
least one significant implementation difference/error that was modified/corrected to 
improve correlation with other simulation tools. 

8.2 Observations 

The following observation was identified: 

0-1. This assessment appears to have been one of the first comparisons of more than two 
atmospheric 6-DOF simulation tools intended for public release. 

8.3 NESC Recommendations 

The following NESC recommendations were identified and directed towards the NESC and/or 
flight simulation tool users as noted. 

R-l. Encourage use of these check-cases to help minimize errors from flight simulation tools. 

(F-ll, F-12) - Flight simulation tool users 
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R-2. Ensure widespread public dissemination and availability of models and results to the 
aerospace community. (F-ll, F-12) - NESC 

R-3. Flat- or round-Earth simplifications should only be used when appropriate. (F-5) - Flight 
simulation tool users 

R-4. Continue development of additional comparisons of missing maneuvers, including 

supersonic maneuvering flight and atmospheric re-entry scenarios. (F-ll) - NESC and 

flight simulation tool users 

R-5. Develop a consensus standard for initial state vector description, employing ANSI/AIAA 
S-l 19-201 1 for identifying simulation parameters, for ease of collaboration and dynamic 
model data exchange. (F-l, F-6) - NESC and flight simulation tool users 

R-6. Develop a consensus standard for time-history data encoding, employing ANSEAIAA S- 
1 19-201 1 for identifying simulation parameters, for ease of collaboration and dynamic 
model data exchange. (F-l, F-6) - NESC and flight simulation tool users 

9.0 Alternate Viewpoint 

There were no alternate viewpoints identified during the course of this assessment by the NESC 
assessment team or the NRB quorum. 

10.0 Other Deliverables 

No unique hardware, software, or data packages, outside those contained in this report, were 
disseminated to other parties outside this assessment. 

11.0 Lessons Learned 

No applicable lessons learned were identified for entry into the NASA Lessons Learned 
Information System (LLIS) as a result of this assessment. 

12.0 Recommendations for NASA Standards and Specifications 

A convention or standard for numerical specification of unambiguous initial conditions for 6- 
DOF flight simulations should be developed to ensure multiple flight simulation tools start at 
exactly the same planet-relative position, velocity, attitude and angular rates. A significant 
portion of this assessment was spent resolving misinterpreted initial conditions despite an 
attempt to specify this information. Questions regarding whether an angular rate initial condition 
was with respect to a rotating Earth or to an inertial axis were raised multiple times, as well as 
ambiguity of initial angular attitude. By way of comparison, the popular two-line element format 
that specifies orbital parameters provides some of this information for satellites, although attitude 
is not included. 
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Also, a structured, binary, compressed format to encode bulky time-history data (which is 
provided for this assessment as comma-separated-value UNICODE text files at the URL 
identified by ref. 7), as well as tools to manipulate this data, should be identified and/or 
developed and adopted by NASA to assist in sharing predicted trajectories from simulation tools. 
Using a CSV format was expeditious but cumbersome. 


13.0 Definition of Terms 


Finding 


Lessons Learned 


Observation 


Problem 

Recommendation 


A relevant factual conclusion and/or issue that is within the assessment 
scope and that the team has rigorously based on data from their 
independent analyses, tests, inspections, and/or reviews of technical 
documentation. 

Knowledge, understanding, or conclusive insight gained by experience 
that may benefit other current or future NASA programs and projects. 

The experience may be positive, as in a successful test or mission, or 
negative, as in a mishap or failure. 

A noteworthy fact, issue, and/or risk, which may not be directly within the 
assessment scope, but could generate a separate issue or concern if not 
addressed. Alternatively, an observation can be a positive 
acknowledgement of a Center/Program/Project/Organization’s operational 
structure, tools, and/or support provided. 

The subject of the independent technical assessment. 

A proposed measurable stakeholder action directly supported by specific 
Finding(s) and/or Observation(s) that will correct or mitigate an identified 
issue or risk. 


14.0 Acronyms List 


AIAA American Institute of Aeronautics and Astronautics 

ANSI American National Standards Institute 

Cd Aerodynamic coefficient of drag 

CSV Comma-separated values 

DAVE-ML Dynamic Aerospace Vehicle Exchange Markup Language 
DoD Department of Defense 

DOF Degree of Freedom 

ECEF Earth-centered, Earth-fixed (rotating coordinate frame) 

ECI Earth-centered inertial (non-rotating coordinate frame) 

EOM Equations of motion 

F Force (thrust) 

h Geometric altitude 
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Ji 

First non-zero gravitational harmonic 

JSC 

Johnson Space Center 

LaRC 

Langley Research Center 

MSFC 

Marshall Space Flight Center 

MET 

Marshall Engineering Thermosphere 

NED 

North-East-Down 

R 

Radius 

S-l 19 

ANSI/AIAA S-l 19-201 1 Flight Dynamic Model Exchange Standard 

T 

Torque 

WGS-84 

World Geodetic System 1984 

XML 

Extensible Markup Language 

Z 

Geopotential height 
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